Display directional sensing

ABSTRACT

An electronic device includes a display with an image display area configured to change orientation based on a change of position of the device. Upon detection of the position change, a camera may capture an image adjacent the device. The orientation of the image display area may be moved relative to a reference feature captured in the image.

BACKGROUND OF THE INVENTION

The present invention relates generally to display devices and, moreparticularly, to display directional sensing of an electronic device.

Electronic devices commonly have a display feature incorporated. Inconventional displays, the orientation of an image displayed may befixed. More recent devices have incorporated a feature that may sensemovement of the host device which may trigger a change in theorientation of a displayed image. For example, a mobile phone or tabletdevice may include auto rotation of an image using a G-sensor.Typically, the auto rotation function may be designed to function whilea user is standing or sitting up right. However, if the device ishorizontal to the ground, the device may become confused in displayingthe orientation of the image may so that it is aligned with the user.

Therefore, it can be seen that there is a need for automaticallycontrolling the orientation of an image on a device regardless of theposition the device is being supported.

SUMMARY

In one aspect, an apparatus comprises a housing; a display on one sideof the housing; a camera coupled to the housing; an orientation sensorcoupled to the housing; a computing device coupled to the orientationsensor; and an image display area on the display controlled by thecomputing device, wherein the computing device is configured to: receivean indication of a change in position of the apparatus provided by theorientation sensor, trigger the camera to capture an image, analyze thecaptured image for a reference feature, and move an orientation of theimage display area to coincide with a position of the reference feature.

In another aspect, a method for displaying an orientation of an imagearea on a display comprises detecting a change in position of anorientation sensor coupled to the display; triggering a camera, upon thedetection, to capture an image of a reference feature adjacent thedisplay; recognizing an orientation position of the reference feature inthe captured image; and rotating a view shown in the image area to alignwith the orientation position of the reference feature.

In a further aspect, a computer readable medium for determining anorientation view of a display on an electronic device comprises acomputer usable program code. The computer usable program code may beconfigured to: detect a change in position of the display; trigger acamera, upon the detected change in position, to capture an image of auser holding the electronic device; enable a facial recognition functionto recognize facial features in the captured image of the user;determine an orientation of the user based on the captured image of theuser; and align a view of an image area in the display to align with theorientation of the user.

These and other features, aspects and advantages of the presentinvention will become better understood with reference to the followingdrawings, description and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a front view of a device with an electronic display and arotatable image area according to an exemplary embodiment of the presentinvention;

FIG. 1B is a block diagram showing a schematic connection of internalelements of the device of FIG. 1A;

FIG. 2A is a front view of the device of FIG. 1A being rotated about anaxis orthogonal to a ground level;

FIG. 2B is an edge view of the device of FIG. 1A being rotated about anaxis planar to a ground level;

FIG. 3A is a perspective view showing the device of FIG. 1A capturing animage of a user holding the device;

FIG. 3B is a front view of the image captured according to FIG. 3A; and

FIG. 4 is a flow diagram of a process of orienting display directionaccording to another exemplary embodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

The following detailed description is of the best currently contemplatedmodes of carrying out exemplary embodiments. The description is not tobe taken in a limiting sense, but is made merely for the purpose ofillustrating the general principles, since the scope of the embodimentsis best defined by the appended claims.

Various inventive features are described below that can each be usedindependently of one another or in combination with other features.

Broadly, exemplary embodiments provide re-orientation of a display imageon an electronic device relative to a reference feature captured in animage taken by the device. A reference alignment may be determined basedon the reference feature. The term “orientation” as used in thisdisclosure may refer to a view of an image area on the display beingmoved or rotated so that its view coincides with a perspective of a userholding the device, for example, so that the image shown in the displayis viewed “right side up” relative to the position of the viewer's eyes.

Exemplary embodiments may take the form of an entire hardwareembodiment, an entire software embodiment (including firmware, residentsoftware, micro-code, etc.) or an embodiment combining software andhardware aspects that may all generally be referred to herein as a“circuit,” “module” or “system.” Furthermore, exemplary embodiments maytake the form of a computer program product embodied in any tangiblemedium of expression having computer-usable program code embodied in themedium.

Any combination of one or more computer usable or computer readablemedium(s) may be utilized. The computer-usable or computer-readablemedium may be, for example but not limited to, an electronic, magnetic,optical, electromagnetic, infrared, or semiconductor system, apparatus,device, or propagation medium. More specific examples (a non-exhaustivelist) of the computer-readable medium would include the following: anelectrical connection having one or more wires, a portable computerdiskette, a hard disk, a random access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM or Flashmemory), an optical fiber, a portable compact disc read-only memory(CD-ROM), an optical storage device, a transmission media such as thosesupporting the Internet or an intranet, or a magnetic storage device.Note that the computer-usable or computer-readable medium could even bepaper or another suitable medium upon which the program is printed, asthe program can be electronically captured, via, for instance, opticalscanning of the paper or other medium, then compiled, interpreted, orotherwise processed in a suitable manner, if necessary, and then storedin a computer memory. In the context of this document, a computer-usableor computer-readable medium may be any medium that can contain, store,communicate, propagate, or transport the program for use by or inconnection with the instruction performance system, apparatus, ordevice. The computer-usable medium may include a propagated data signalwith the computer-usable program code embodied therewith, either inbaseband or as part of a carrier wave. The computer usable program codemay be transmitted using any appropriate medium, including but notlimited to wireless, wired, wire line, optical fiber cable, RF, etc.

Computer program code for carrying out operations of exemplaryembodiments may be written in any combination of one or more programminglanguages, including an object oriented programming language such asJava™, Smalltalk™, C++ or the like, and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computing device, partly on the user's computing device, as astand-alone software package, or partly on the user's computing deviceand partly on a remote computing device.

Exemplary embodiments are described below with reference to flowchartillustrations and/or block diagrams of methods, apparatus (systems) andcomputer program products. It will be understood that each block of theflowchart illustrations and/or block diagrams, and combinations ofblocks in the flowchart illustrations and/or block diagrams, can beimplemented by computer program instructions.

These computer program instructions may be provided to a processor of ageneral purpose computer, special purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions, which execute via the processor of the computingdevice or other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

These computer program instructions may also be stored in acomputer-readable medium that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablemedium produce an article of manufacture including instruction meanswhich implement the function/act specified in the flowchart and/or blockdiagram block or blocks. The computer program instructions may also beloaded onto a computer or other programmable data processing apparatusto cause a series of operational steps to be performed on the computeror other programmable apparatus to produce a computer implementedprocess such that the instructions which execute on the computer orother programmable apparatus provide processes for implementing thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

Referring to FIGS. 1A and 1B, an electronic display device 100 is shownaccording to an exemplary embodiment of the present invention. Thedisplay device 100 may be, for example, selected from a group consistingof a game controller, a portable gaming device, a cell phone, a camera,a tablet, a laptop, a personal digital assistant (PDA), a portable musicplayer, a portable video player, a remote control, a digital photoframe, a camcorder, a portable television, a global positioning system(GPS) device, a smart phone, a network base station, a media player, anavigation device, an e-mail device, a digital-video-recorder (DVR), asecurity system (e.g., a door or gate access system), or a combinationof any two or more of these data processing devices or other dataprocessing devices. The display device 100 may comprise any type ofelectronic device, general purpose computing device or special purposecomputing device that includes a processor, other circuitry, or logicoperable to perform a screen switch process described herein tofacilitate a user's view of an image on display.

In an exemplary embodiment, the display device 100 may include a housing101, a screen display 102, a camera 106, and an image display area 109viewable from an exterior of the display device 100. In someembodiments, an infra-red sensor 124 may be included on the housing 101exterior. The display device 100 may also include an orientation sensor112, a memory module 118, and a computing device 120. The orientationsensor 112, the memory module 118, and the computing device 120 may belocated within the housing 101. Computer readable code and files 122 maybe stored in some combination between the memory 118 and the computingdevice 120. In an exemplary embodiment, the computing device 120 may becoupled to and configured to receive data from and control the display102, the camera 106, the orientation sensor 112, the memory module 118,and the infra-red sensor 124. In some embodiments, the camera 106 may becontinuously operated. In some embodiments, the camera 106 may beoperated only once a triggering event occurs. The computing device 120may be for example, a processor, a central processing unit (CPU), orother computing apparatus.

The display 102 may be implemented with liquid crystal display (LCD)technology, light emitting polymer display (LPD) technology, or someother display technology. The display 102 may be configured to respondto haptic and/or tactile contact by a user. In some implementations, thedisplay 102 may comprise a multi-touch-sensitive display. Amulti-touch-sensitive display may, for example, process multiplesimultaneous touch points, including processing data related to thepressure, degree, and/or position of each touch point. An image displayarea 109 in the display 102 may be configured to show images, forexample, in portrait or landscape view orientations or other angles inbetween the portrait position and landscape position. In an exemplaryembodiment, the image display area 109 may rotate the view of an imageas a user moves or rotates the device display 100. In an exemplaryembodiment, the computing device 120 may determine the orientation ofthe image display area 109 relative to the user and may change theorientation of the image displayed accordingly.

The orientation sensor 112 may be for example, an accelerometer, aG-sensor, or a rotational sensor (for example, a gyroscopic basedsensor). In some embodiments, the orientation sensor 112 may be adual-axis accelerometer. In some embodiments, the orientation sensor 112may be a tilt sensor, an electronic compass or other suitable sensors,or combinations thereof configured to detect a position change of theorientation sensor 112 with respect to, for example, gravity or ahorizontal ground plane reference. In some embodiments, the displaydevice 100 may comprise two or more sensors, such as an accelerometerand an electronic compass.

As will be appreciated by persons skilled in the art, an accelerometeris a sensor which converts acceleration from motion (e.g., movement ofthe display device 100 or a portion thereof due to a strike force) andgravity which are detected by a sensing element into an electricalsignal (producing a corresponding change in output) and may be availablein one, two or three axis configurations. Accelerometers may producedigital or analog output signals depending on the type of accelerometer.Generally, two types of outputs may be available depending on whether ananalog or digital accelerometer is used: (1) an analog output requiringbuffering and analog-to-digital (A/D) conversion; and (2) a digitaloutput which is typically available in an industry standard interface,such as an SPI (Serial Peripheral Interface) or I2C (Inter-IntegratedCircuit) interface.

The output of an accelerometer is typically measured in terms of thegravitational acceleration constant at the Earth's surface, denoted by“g”, which is approximately 9.81 m/s² (32.2 ft/s²) as the standardaverage. The accelerometer may be of various types including, but notlimited to, a capacitive, piezoelectric, piezoresistive, or gas-basedaccelerometer. The range of accelerometers may vary up to the thousandsof g's, however, for portable electronic devices, “low-g” accelerometersmay be used. Examples of low-g accelerometers which may be used aremicro electro-mechanical systems (MEMS) digital accelerometers fromAnalog Devices, Inc. (ADI), Freescale Semiconductor, Inc. (Freescale)and STMicroelectronics N.V. of Geneva, Switzerland.

Referring now to FIGS. 2A and 2B, exemplary positioning of the displaydevice 100 and detection of its movement by the orientation sensor 112is shown relative to a reference point. It will be understood thatreference to internal components of the display device 100 as describedin FIG. 1A may be present while describing features of the presentinvention. In an exemplary embodiment, the orientation sensor 112 maydetect a position change of the display device 100. The orientationsensor 112 may detect when the sensor has been moved with respect to aground reference plane “G” or rotated about an axis relative to theground reference plane “G”.

FIG. 2A shows the display device 100 being rotated about an axis 260orthogonal to a ground level, for example, the ground reference plane“G”. The axis 260 may point in the direction of gravity. For a dual-axisorientation sensor 122, a measurement axis 280 may define a first “Y”axis through the device 100 running, for example, from a top edge 250 ofthe device 100 down through a bottom edge 256. The axis 280 may beparallel to a side edge 252 and 254. A second “Y” axis 270 may run alongthe side edge 252 parallel to the first “Y” axis 280. “X” axes may beorthogonal to the “Y” axes 270 and 280. The “X” axes may run, forexample, parallel to the top and bottom edges 250 and 256. The display102 may face outward along a “Z” axis. The ‘Z” axis may be orthogonal tothe “X’ and “Y” axes and parallel to the ground reference plane “G”. Theaxis 280 may be aligned with the axis 260 when, for example the top edge250 is parallel to the ground reference plane “G” and the device 100 maybe considered in an upright position displaying, for example, a“portrait” orientation on the image display area 109. When the display102 is rotated, for example clockwise as shown, the computing device 120may read the amount of movement the measurement axis 280 has moved fromthe axis 260.

Movement may be measured by change in angular position, for example, asin Euler angles. Movement and change in position may also be referred toin changes of the display device's 100 yaw, pitch, or roll. In anexemplary embodiment, the amount of movement measured, for example, themagnitude of an angle α between the measurement axis 280 and the axis260 may be represented by the symbol Δ. In an exemplary embodiment, thememory 118 may store a threshold value for angle α. When Δ exceeds thethreshold value for angle α, the computing device 120 may trigger use ofthe camera 106 to capture an image.

While movement has been described with respect to the “Y” axis 280, itwill be understood that the measured Δ may be calculated for rotationabout any of the axes running through the display device 100, shown orotherwise. For example, while the foregoing has been described withmovement of the device 100 being performed orthogonal to the groundreference plane “G” in a clockwise rotation, it will be understood thatother changes in position of the device 100 may be detected by theorientation sensor 112. Referring to FIG. 2B, the display screen 102 mayinitially be planar to the ground reference plane “G”. The ‘Y” axisremains pointing toward gravity and the ground reference plane but thedisplay 102 faces the “Y” axis while the “Z” axis runs through the topedge 250 and bottom edge 256 and is planar to the ground reference plane“G”. The orientation sensor 112 may detect rotation of the displaydevice 100 around the “Z” axis.

Referring now to FIGS. 3A and 3B, an exemplary operation of the device100 is shown according to an exemplary embodiment of the presentinvention. The device 100 is being held by the hand “H” of a user. FIG.3A illustrates the device 100 being moved so that the orientation sensor112 (FIG. 1B) has triggered the camera 106. The camera 106 may bedisposed to capture an image of a reference feature adjacent the display102. In an exemplary embodiment, the camera 106 may capture an image 312of the user 302. The computing device 120 may be configured to recognizean orientation position of the reference feature in the captured image.For example, the device 100 may include facial recognition softwareconfigured to recognize facial features present in the image of theuser's face captured by the camera 106. In an exemplary embodiment, thefeatures of the face recognized may include a nose 308, a mouth 305, anda set of eyes (306 _(L) and 306 _(R)) (referred to generally as eyes306). In another exemplary embodiment, the position of the eyes may bedetected by the infra-red sensor 124 (FIG. 1B). The computing device maybe configured to determine an axis 304 across the eyes 306 _(L) and 306_(R). The nose 308 and mouth 305 may be detected as located below theaxis 304. The computing device 120 may re-create the axis 304 onto theimage display area 109 as an axis 310. The computing device 120 maydetermine the position of the nose 308 and/or the mouth 305 relative tothe eyes 306 _(L) and 306 _(R) in deciding the position of the axis 310.The orientation of the image 312 may be moved or rotated to alignperpendicular with the axis 310 so that the position of the eyes 306along with the nose 308 and/or mouth 305 in the captured image 312 alignwith the position of the user's 302 eyes, nose and/or mouth.

In general, a facial analysis system present on the display device 100may be defined in a computer application, hardware, or a combination ofhardware and software, for automatically identifying or verifying aposition of a person's head from a digital image or a video frame from avideo source. In one embodiment, facial reference features captured fora user 302 may be stored in a database, for example, in memory module118. The database may be coupled to logic that allows identification ofspecific facial features, while allowing for variations in expectedplacement of features. A user 302 may be identified relatively quicklyso orientation of the image display area 109 may be performed quicker.Facial recognition algorithms may identify a user 302 by extractinglandmarks or features from an image of the subject's face. For example,an algorithm may analyze the relative position, size, or shape of theeyes 306, nose 308, mouth 305, cheekbones, jaw line, forehead, chin, etcwithin a captured image 312. These features may then be used to searchfor other stored images with matching features or to calculateprobabilities that specific human features are present, and then tocalculate their location within the image. In still another embodiment,an algorithm may be used to normalize a gallery of captured face images312 and may compress the face related data, only saving the data in theimage that is useful for face detection. This will provide face metadatathat can be quickly indexed and searched.

In another exemplary embodiment, in addition to face recognition,objects on or around the user 302 may be used to detect orientation. Forexample, arbitrary objects may be recognized and their capturedorientation may be checked against a database of files that may includean expected orientation. For example, the camera 106 may capture a chairin the background behind the user 302. The computing module 120 mayrecognize the shape of the chair and may check the memory module 118 forthe expected orientation direction of the chair within the image displayarea 109. If the chair appears in the captured image 312 in a directioninconsistent with the expected orientation, the computing device 120 mayrotate the image display area 109 so that the chair is displayed in itsexpected orientation. Other arbitrary objects captured within the image312 may include items worn by the user 302. For example, glasses, hats,ear buds, headphones, shirt collars, necklaces, hair pins, etc. may berecognized and the computing device 120 may determine their expectedorientation similar to the example provided using a chair. These objectsalone or used in conjunction with detected facial features may be usedto detect orientation and/or position of the image display area 109relative to the orientation of the user's face.

Facial recognition may be performed at different frequencies of imagecapture. For example, face recognition may be performed at each videoframe, every 10 frames, every second, etc. The frequency of facialrecognition analysis may depend on the amount of processing required andthe responsiveness of the display device 100. In one embodiment, theview orientation may not change unless several frames have been detectedrequiring a change in view orientation. This may avoid the problem ofswitching views when the user is playing a game and performs an abruptmovement of the device before returning to the previous position,unintentionally triggering the display device's 100 mechanisms forchanging the orientation of the image display area 109. In anotherembodiment, the facial recognition frequency may be increased when thecomputing device 120 detects frequent movement of the user 302 holdingthe display device 100 so that a change in display image area 109orientation may be needed for consistent viewing.

Referring now to FIG. 4, a method 400 for displaying an orientation ofan image area on a display is shown according to an exemplary embodimentof the present invention. The computing device 120 may receive datadetecting (402) a change in position of the orientation sensor. Thecomputing device 120 may determine (404) if Δ exceeds the thresholdvalue for angle α. The computing device 120 may trigger (406) operationof the camera 106 when the threshold value is exceeded. In someembodiments, the camera 106 may be continuously on while in someembodiments, the camera 106 may be dormant until the computing device120 commands the camera 106 to capture an image. Otherwise, if thethreshold value is not exceeded, the computing device 120 waits until itreceives data detecting (402) a change in position of the orientationsensor 112 again. The computing device 120 may capture (408) an imagewith the camera 106. In some embodiments, the computing device 120 maycease (410) operation of the camera 106 to conserve power in the displaydevice 100. The computing device 120 may perform (412) imaging analysisof the captured image. The computing device 120 may determine (414) anorientation direction for the image display area 109 based on theimaging analysis. The orientation direction may be aligned with areference feature detected in the captured image. The computing device120 may determine (416) if the determined orientation direction of theimage display area 109 is different than an orientation directioncurrently displayed within the image display area 109. If they differ,then the image display area 109 may be rotated to the determinedorientation direction. Otherwise, the computing device 120 waits untilit receives data detecting (402) a change in position of the orientationsensor 112 again.

It should be understood, of course, that the foregoing relate toexemplary embodiments of the invention and that modifications may bemade without departing from the spirit and scope of the invention as setforth in the following claims.

We claim:
 1. An apparatus, comprising: an image display area; a camera;an orientation sensor in communication with the camera; a computingdevice coupled to the orientation sensor and the image display area; andwherein the computing device is configured to: receive an indicationfrom the orientation sensor of a change in position of the apparatuswhich change exceeds a threshold value, turn the camera on and capturean image; analyze the captured image for a reference feature that isunassociated with a user's face and head, orient the image display areain accordance with the reference feature, and turn the camera off untilreceiving another indication of another change in position of theapparatus that exceeds the threshold value.
 2. The apparatus of claim 1,wherein the computing device controls the image display area to switchbetween a landscape orientation and a portrait orientation.
 3. Theapparatus of claim 1, wherein the camera is on the same side as theimage display area.
 4. The apparatus of claim 1, comprising an infra-redsensor coupled to the computing device.
 5. The apparatus of claim 1,wherein the orientation sensor is configured to detect movement of theapparatus so that the image display area is planar to a ground level. 6.The apparatus of claim 5, wherein the camera is triggered by theorientation sensor sensing a change in the position of the apparatusrelative to the ground level, wherein the change is greater than thethreshold level.
 7. A method for orienting an image display area of adisplay device, comprising: detecting a threshold change in position ofan orientation sensor of the display device; turning on a camera of thedisplay device upon detecting the threshold change in position of theorientation sensor; capturing with the camera an image of a referencefeature adjacent the display; recognizing an orientation position of thereference feature in the captured image; and rotating a view shown inthe image display area to align with the orientation position of thereference feature; and turning the camera off until detecting anotherthreshold change in position of the orientation sensor.
 8. The method ofclaim 7 further comprising determining if a direction of the view shownin the image display area is different than a direction of theorientation position of the reference feature.
 9. The method of claim 7further comprising recognizing a face of a user holding the displaydevice, wherein the reference feature is on the face.
 10. The method ofclaim 9 further comprising determining a relative position of thereference feature on the face to other reference features on the face.11. The method of claim 10 further comprising determining an axis alongthe face and the orientation position of the reference feature based onthe determined axis.
 12. A computer readable medium for determining anorientation view of a display on an electronic device having computerusable program code embodied therewith, the computer usable program codeconfigured to: detect a change in position of the display; determine ifthe change in position exceeds a threshold value; turn on a cameracoupled to the display if the threshold value is exceeded; use thecamera to capture an image of a user of the electronic device; enable afacial recognition function to recognize facial features in the capturedimage of the user; determine an orientation of the user based on thecaptured image of the user; align a view of an image display area in thedisplay to align with the orientation of the user; and turn off thecamera if the threshold value is not exceeded.
 13. The computer programcode of claim 12 further comprising computer program code configured todetermine positioning of the facial features and determine theorientation of the user based on the positioning of the recognizedfacial features.
 14. The computer program code of claim 13 furthercomprising computer program code configured to determine an axis of theuser's face based on the positioning of the facial features and alignthe axis of the face with an axis of the image display area.
 15. Thecomputer program code of claim 13 further comprising computer programcode configured to determine a position of a nose on the user's facerelative to a position of eyes on the face and align a top of the viewof the image display area to the display to the user's eyes.
 16. Thecomputer program code of claim 12 further comprising computer programcode configured to turn off the camera after capturing the image of theuser holding the electronic device.